Skip to content

feat: backend quota and auto-compaction spec fields, NOSPACE alarm surfacing#401

Open
xrl wants to merge 6 commits into
etcd-io:mainfrom
xrl:pr/db-size-lifecycle
Open

feat: backend quota and auto-compaction spec fields, NOSPACE alarm surfacing#401
xrl wants to merge 6 commits into
etcd-io:mainfrom
xrl:pr/db-size-lifecycle

Conversation

@xrl

@xrl xrl commented Jul 7, 2026

Copy link
Copy Markdown

What

Database-size lifecycle basics for EtcdCluster, in two halves:

  1. Spec knobs for quota and compaction. Three new optional fields — spec.quotaBackendBytes (resource.Quantity), spec.autoCompactionMode (enum periodic/revision), spec.autoCompactionRetention (pattern-validated duration or revision count) — render the matching --quota-backend-bytes / --auto-compaction-mode / --auto-compaction-retention container args. Flags are emitted only when set, so existing clusters keep byte-identical args and nothing rolls on operator upgrade; a conflicting flag in etcdOptions still wins, following the documented precedence. A CEL rule rejects autoCompactionMode without autoCompactionRetention.

  2. NOSPACE alarm surfacing. Without a quota story the operator is silent at the worst moment: when etcd hits its backend quota it raises a NOSPACE alarm and the cluster goes read-only, while the status only shows the generic UnhealthyMembers. The reconciler now polls AlarmList after the member health check (even when the health check itself errored — an alarmed member reports unhealthy, and the deferred status update still runs). An active NOSPACE alarm sets the Degraded condition with distinct reason DatabaseQuotaExceeded and emits a Warning event per affected member with the operational runbook (compact, defragment, etcdctl alarm disarm; a raised quotaBackendBytes only lands on the next StatefulSet rollout). Detection only — no defrag or disarm automation.

Files

  • api/v1alpha1/etcdcluster_types.go — three fields + CEL cross-field rule.
  • internal/controller/utils.godefaultArgs/createArgs take the EtcdCluster and render the flags conditionally.
  • internal/etcdutils/etcdutils.goAlarmList + MemberAlarm (carries the alarm type string, so CORRUPT is one branch away for a follow-up), mirroring the existing client boilerplate.
  • internal/controller/etcdcluster_controller.go — alarm poll in performHealthChecks, DatabaseQuotaExceeded condition reason in updateConditions (takes precedence over UnhealthyMembers), per-member Warning event, events.k8s.io RBAC marker.
  • docs/configuring-etcd-options.md — new "Database size and compaction" section, including the rollout-timing caveat.
  • config/crd/**, config/rbac/role.yaml, docs/api-references/docs.md, zz_generated.deepcopy.go — regenerated via make generate manifests api-docs.

Review

An adversarial review pass produced 2 blocking/important findings, both resolved in follow-up commits: the mgr.GetEventRecorder recorder emits events.k8s.io/v1 Events, so the core-group events RBAC rule alone left the DatabaseQuotaExceeded event forbidden at runtime (blocking — dedicated rule added); and the event/docs wording implied raising spec.quotaBackendBytes takes effect immediately, but steady-state reconciles never patch the pod template (important — reworded to "next StatefulSet rollout" in both places). Remaining notes were minor or deliberately descoped (below).

Testing

  • TestCreatingArgs (extended table): quota rendering (8Gi--quota-backend-bytes=8589934592), compaction mode+retention, unset fields emitting no flags, etcdOptions conflict precedence.
  • TestUpdateConditionsNospaceAlarm: NOSPACE sets the distinct DatabaseQuotaExceeded reason; unhealthy-without-alarm keeps the generic reason.
  • TestReportNospaceAlarms: Warning event emitted with the member name resolved from the member list; nil recorder does not panic; non-NOSPACE alarms emit nothing.
  • TestAlarmList (internal/etcdutils, against a live embedded server): no active alarms, error on invalid endpoint.
  • Full make test and make verify pass.

Descoped follow-ups

(1) No kubebuilder defaults for the new fields, deviating from the brief: CRD-defaulting quota/compaction to etcd's defaults would inject new container args into every existing StatefulSet and roll all clusters on operator upgrade; flags render only when set, and etcd's defaults (2GiB quota, 8GiB recommended max) are documented in the field comments. (2) Defrag automation, alarm disarm, and any LastDefragTime status field — explicit follow-up per the brief. (3) CORRUPT alarm surfacing — etcdutils.MemberAlarm carries the type string so a follow-up is one condition branch, but this PR reacts to NOSPACE only. (4) Numeric CEL bounds on quotaBackendBytes (e.g. <= 8Gi) — quantity() CEL comparisons are version-sensitive and etcd only warns above 8GiB; comment-documented instead. (5) No per-member status field for alarms and no new status API — condition + event only. (6) No sample-manifest or e2e additions (the test/e2e/etcd_options_test.go pattern would need kind); unit/fake coverage only, matching #382-sized PR precedent. (7) TLS-aware alarm client — the etcdutils helpers, including the existing healthCheck path, are already plaintext-only on origin/main; AlarmList follows the same pattern, and fixing TLS client wiring is a separate gap.

🤖 Generated with Claude Code


PR series — operability fixes, TLS & EtcdMirror

Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.

PR Lands After
Reviewable now — small, any order
🟢 #374 Requeue instead of swallowing the client-cert provisioning error
🟢 #391 Grant events.k8s.io RBAC so operator Events are actually recorded
🟢 #392 Correlate members[]/leaderID from one health snapshot (consistent leader)
🟢 #393 Propagate user-supplied altNames.ipAddresses into certificates
🟢 #394 Accept day-suffix validityDuration (365d, 100d12h) as documented
🟢 #395 Surface early reconcile errors as a Degraded condition (was empty status)
🟢 #379 Configurable reconcile worker pool (--max-concurrent-reconciles)
🟢 #369 kind-based stress/scale e2e harness (1/3/7 members, churn, quorum watcher)
🟢 #398 podTemplate.spec scheduling fields: topologySpread, resources, priorityClass, schedulerName
🟢 #397 First-class Helm chart covering the full config/ surface
🟢 #399 Quorum-aware PodDisruptionBudget per EtcdCluster
🟢 #400 Scale-in removes the right member and transfers leadership first
🟢→ #401 Backend quota + auto-compaction spec fields, NOSPACE alarm surfacing
🟢 #402 Reconcile-loop gates replace the blocking StatefulSet-ready wait
🟢 #403 Quorum-gated one-pod-at-a-time version upgrades via OnDelete
TLS stack — in order
🟢 #376 Independent spec.tls.{peer,client} surfaces (breaking alpha API)
#377 TLSReady condition + TLS lifecycle Events #376
#378 Multi-member TLS quorum e2e + PeerCANotShared #377
EtcdMirror — in order
🟢 #406 EtcdMirror CRD: one-way cross-cluster replication API (types + CEL)
🟢 #407 pkg/mirroragent replication engine (fenced checkpoint-in-target) #406
🟢 #408 Periodic reconciliation pass wired into the engine's steady-state loop #407
🟢 #412 mirror-agent binary: config/TLS-reload/statusz/metrics + kind e2e #408
🟢 #413 EtcdMirror controller: Deployment rendering, statusz-driven conditions, guards, fenced finalizer #412
🟢 #414 CRD/sample/editor+viewer-role kustomize wiring #413
🟢 #415 Controller-side phase/condition gauges, shipped PrometheusRule + dashboard #414
🟢 #416 Full-lifecycle kind e2e: fence overlap, compaction+prune, restart resume, cert rotation, cutover/reversal #415
Parked as drafts pending #363
#382 Per-cluster domain metrics on the operator /metrics endpoint
#384 EtcdCluster admission webhooks (consolidating with #328) #363
#386 EtcdBackup CR → object storage (S3/GCS) #363
#387 Automatic quorum-loss disaster recovery (bootstrap-latch guarded) #363

🟢 ready · ⚪ draft

xrl and others added 4 commits July 7, 2026 04:00
spec.quotaBackendBytes, spec.autoCompactionMode and
spec.autoCompactionRetention render the corresponding etcd flags.
Flags are emitted only when set so existing clusters keep
byte-identical args; etcdOptions still wins on conflict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
Poll etcd alarms after the member health check. An active NOSPACE
alarm sets Degraded reason DatabaseQuotaExceeded (overriding the
generic UnhealthyMembers) and emits a Warning event per member.
Detection only: no defrag or disarm automation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
mgr.GetEventRecorder emits events.k8s.io/v1 Events; the core-group
events rule alone leaves the DatabaseQuotaExceeded event forbidden.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
Steady-state reconciles never patch the pod template, so the NOSPACE
event and docs must not imply raising spec.quotaBackendBytes is
applied immediately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign jberkus for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

This was referenced Jul 7, 2026
xrl added 2 commits July 8, 2026 03:22
- quotaBackendBytes below 100Mi (CEL quantity floor): etcd disables the
  quota for non-positive values, and a byte-scale typo ("8" for "8Gi")
  raises a cluster-wide NOSPACE on the first pod restart.
- a non-integer autoCompactionRetention in revision mode: etcd falls
  back to time.ParseDuration and uses the raw nanosecond count as the
  revision retention, so compaction effectively never runs.

Regenerated CRD, chart CRD and API docs.

Signed-off-by: Xavier Lange <xrlange@gmail.com>
(cherry picked from commit 785e8d7)
…rollout'

The operator re-renders the template on any spec edit and starts a
quorum-gated one-pod-at-a-time restart at once; under an active NOSPACE
alarm the health gates hold it until the alarm is disarmed. Fix the doc
and event text, and add the operator-downgrade NOSPACE caveat.

Signed-off-by: Xavier Lange <xrlange@gmail.com>
(cherry picked from commit eefd979)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant